TIP SHEET Y) Srional cone 


Effectiveness in Context: Tips for Identifying Evidence - Based 
Practices for Diverse Student Populations 


While finding interventions with proven effectiveness has become easier through resources like the 
What Works Clearinghouse (WWC), identifying what works for diverse student populations can still 
be challenging. Knowing the context, specifically, sample demographics for research, can help 
bridge this gap. In a recent scan of studies that met WWC standards with and without reservations, 
and that had at least one statistically significant positive finding, 70 percent of the resulting WWC 
reviews reported race for at least a segment of the study sample. Decision-makers can use this 
information, when available, to help decipher how effective an intervention might be for their 
specific students. This tip sheet explains the importance of the demographic information provided 
in research studies and provides steps to identify evidence-based practices and the sample 
characteristics behind the evidence. 


Tip 1: Understanding the demographic composition of study samples 
can help educators determine whether a practice is evidence-based 
for their student populations. 


» Sample demographics tell the important story 
of how generalizable outcomes are to districts 
and schools. For the 70 percent of reviews in 
our scan that reported race for at least a 
segment of the study sample, more than half aR SERS 
of the reviews had a sample that was Racial Group 


predominately (greater than 50%) one racial m One Predominant 
Racial Group 


Percentage of reviews with a 
predominant racial group 


group. This finding may cause educators to 
wonder if an intervention has been proven 
effective for the students they serve. 


» More specifically, of the reviews included in the scan that reported race, 28 percent reported a 
study sample greater than 50 percent White. Of those, 9 percent reported a study sample 
greater than 75 percent White. For 24 percent of reviews in this scan, the study sample was 
greater than 50 percent Black. Of those, 5 percent reported a study sample greater than 
75 percent Black. Only one review in this scan reported a study sample greater than 50 percent 
Asian, with a reported study sample, more specifically, 100 percent Asian. With respect to 
ethnicity, 19 percent of reviews in this scan reported a study sample greater than 50 percent 
Hispanic. Of those, 2 percent were entirely Hispanic. 
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» No reviews in this scan reported a study sample greater than 50 percent Native American or 
Pacific Islander. The highest study sample reported as Pacific Islander was 41 percent, and the 
highest study sample reported as Native American was 14 percent. 


» These characteristics are not always available in a WWC review. Some studies do not report this 
information, while others do not report it for the sample of students who were analyzed for 
study outcomes. In the next two tips, you will learn how to locate this important information 
when it is available in WWC reviews. 


Tip 2: Finding and comparing evidence-based practices is a key step 
toward assessing effectiveness in context. 


» The WWC has been a notable resource for identifying evidence-based programs and practices 
since 2002. There are different ways to find information on “what works” through the WWC. 


» The Find What Works tool allows you to compare the evidence of effectiveness of interventions 
by topic and grade (see Exhibit 1). First select a topic of interest, and then select a single 
intervention to view or compare several interventions. 


Exhibit 1. Use Find What Works and select an intervention to view the underlying 
evidence 
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» After selecting an intervention, you can view citations and review details for the studies, meeting 
WWC standards, with and without reservations. You can also export review results across 
intervention reports to an Excel file (as displayed above in Exhibit 2). Once an exported file is 
downloaded, the file provides you access to review details across intervention reports, including 
the intervention delivery method, demographics, including race and ethnicity, and the 
effectiveness rating by outcome. 


Exhibit 2. Select a citation to see review details or export results 
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Reading Recovery® 


Reading Recovery® is a short-term tutoring intervention that provides one-on-one tutoring to first-grade students who are struggling in 
reading and writing. The goals of Reading Recovery® include promoting literacy skills, reducing the number of students who are 
struggling to read, and preventing long-term reading difficulties. Reading Recovery® supplements classroom teaching with tutoring 
sessions, generally conducted as pull-out sessions during the schoo! day. Tutoring is delivered by trained Reading Recovery teachers in 
daily 30-minute sessions over the course of 12-20 weeks. 


Reviewed Research 
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» The Reviews of Individual Studies feature is useful for larger searches across WWC products, 
including intervention reports, practice guides, and reviews of individual studies. This feature 
was used for the scan discussed in the first tip. To use this feature, first select any filters of 
interest. Then, select a citation to see review details for each search result. 
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» Filters allow you to narrow your search by WWC study rating, topic, study design, or Every 
Student Succeeds Act (ESSA) evidence tier (see Exhibit 3). 


Exhibit 3. Select a citation under Reviews of Individual Studies to view study 
details 


a] 
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© REVIEWS OF INDIVIDUAL STUDIES Searching for WWC’s Introduction to the 
Reviews of Individual WWC's Reviews of 
Studies Individual Studies 


Use this search page to find individual studies that have been reviewed by the WWC and categorized into ESSA evidence tiers. Select 
options from the filters or enter author/title information into the search box. The resulting list of studies connects you to more information, 
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Abelman, R., & Molina, A. (2001). § 
1(1-2), 32-39. 


le over substance revisited: A longitudinal analysis of intrusive intervention. NACADA Journal, 


Agodini, R., Harris, B., Seftor, N., Remillard, J., & Thomas, M. (2013). After two years, three elementary math curricula outperform a 
fourth (NCEE 2013-4019). Washington, DC: National Center for Education Evaluation and Regional Assistance, Institute of 
Education Sciences, U.S. Department of Education. 
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» Similar to the export feature available using the Find What Works tool, the Reviews of Individual 
Studies feature also allows you to download results to an Excel file (see Exhibit 4). While the 
feature appears only when you select an individual study citation, the exported file includes 
reviewed studies across products. You can then filter the file to examine review details based 
on WWC study rating, topic, study design, ESSA evidence tier, and sample characteristics 
(e.g., grade, gender, ethnicity, and race). 


Exhibit 4. Examine details for a single study or export results 
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Abelman, R., & Molina, A. (2001). NACADA Journal, 21(1-2), 32-39. 
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This review may not reflect the full body of research evidence for this intervention. 


Tip 3: Examining the sample characteristics behind the evidence is the 
other pivotal step for assessing effectiveness in context. 


» As discussed in the first tip, the majority of reviews in our scan included details on race for at 
least a segment of the population. At the same time, the scan showed that many reviews had 
study samples that were overrepresented by one racial group. Educators can use these details 
to assess how relevant outcomes are to their students. 


» To find sample characteristics, first select an individual study citation (through the Find What 
Works tool or the Reviews of Individual Studies feature). Then, select the sample characteristics 
tab. Under this tab (as displayed in Exhibit 5), you can view available sample characteristics as 
classified in the study. These characteristics also appear in exported files obtained by using the 
export icon at the top of the page. 
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Exhibit 5. Examine sample characteristics under each study citation or export them 
IES > WWC Stiime Suen + Searct Go 


‘e) WWC REVIEW OF THIS STUDY © Export © Print 


After two years, three elementary math curricula outperform a fourth (NCEE 
2013-4019). 


Agodini, R., Harris, B., Seftor, N., Remillard, J., & Thomas, M. (2013). Washington, DC: National Center for Education Evaluation and 
Regional Assistance, Institute of Education Sciences, U.S. Department of Education. Retrieved from: https://eric.ed.gov/? 
id=ED544185 
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Characteristics of study sample as reported by study author. 
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Connecticut, Florida, Kentucky, 
Minnesota, Missouri, Mississippi, 
Nevada, New York, South Carolina, Texas 


Obtaining the necessary information to make decisions in context can be accomplished using the 
tips above. However, when sample characteristics are limited or unavailable in the research, 
educators will have more of a challenge in making informed decisions about which educational 
interventions to use with their students. Rather than using racial or ethnic demographic data, they 
may look to other available sample characteristics like gender, percentage of English language 
learners, or student free or reduced-price lunch status. They may also look to other study 
characteristics like geographic location or community setting, which also provide context for study 
findings. Researchers, along with state and local school districts, can also contribute to providing 
this vital context to aid decision-making. They can be intentional about including demographic data 
and outcomes for subgroups (where applicable) in studies that their evaluation teams design. 
These deliberate efforts will strengthen the evidence base on which significant decisions are made. 


Contributor: Sakari Morvey 
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